Information Bottlenecks, Causal States, and Statistical Relevance Bases: How to Represent Relevant Information in memoryless transduction

Recently, Tishby, Pereira, and Bialek proposed a new method for finding concise representations of the information one set of variables contains about another [1]. This brief note explains the connections between this “information-bottleneck” method and existing mathematical frameworks and techniques. This comparison should enhance the value of research in this promising direction and clarify the relative uses of these techniques in applications. In the interest of space, we assume readers are familiar with the notation of both [1] and [2].